Rendering responses to a spoken utterance of a user utilizing a local text-response map

ABSTRACT

Implementations disclosed herein relate to generating and/or utilizing, by a client device, a text-response map that is stored locally on the client device. The text-response map can include a plurality of mappings, where each of the mappings define a corresponding direct relationship between corresponding text and a corresponding response. Each of the mappings is defined in the text-response map based on the corresponding text being previously generated from previous audio data captured by the client device and based on the corresponding response being previously received from a remote system in response to transmitting, to the remote system, at least one of the previous audio data and the corresponding text.

BACKGROUND

Voice-based user interfaces are increasingly being used in the controlof computers and other electronic devices. One particularly usefulapplication of a voice-based user interface is with portable electronicdevices such as mobile phones, watches, tablet computers, head-mounteddevices, virtual or augmented reality devices, etc. Another usefulapplication is with vehicular electronic systems such as automotivesystems that incorporate navigation and audio capabilities. Suchapplications are generally characterized by non-traditional form factorsthat limit the utility of more traditional keyboard or touch screeninputs and/or usage in situations where it is desirable to encourage auser to remain focused on other tasks, such as when the user is drivingor walking.

Voice-based user interfaces have continued to evolve from earlyrudimentary interfaces that could only understand simple and directcommands to more sophisticated interfaces that respond to naturallanguage requests and that can understand context and manageback-and-forth dialogs or conversations with users. Many voice-baseduser interfaces incorporate both an initial speech-to-text conversionthat converts an audio recording of a human voice to text, and asemantic analysis that analysis the text in an attempt to determine themeaning of a user's request. Based upon a determined meaning of a user'srecorded voice, an action may be undertaken such as performing a searchor otherwise controlling a computer or other electronic device.

A user may submit queries and/or commands to a client device via aspoken utterance, verbally indicating what information the user hasinterest in being provided and/or an action that the user has interestin being performed. Typically, the spoken utterance is detected bymicrophone(s) of the client device and captured as audio data. The audiodata is transmitted to a remote system for further processing. Theremote system processes the audio data to determine an appropriateresponse, and transmits the response to the client device for renderingby the client device.

Processing of audio data by a remote system can include using aspeech-to-text (STT) component to generate text based on the audio data,where the generated text reflects a spoken utterance captured by theaudio data. The processing can further include processing the generatedtext using a natural language processor (NLP) and/or other semanticprocessor, in an attempt to determine the meaning or intent of thetext—and an action to be performed based on the determined meaning. Theaction can then be performed to generate a corresponding response, andthe corresponding response transmitted to the client device from whichthe audio data was received.

Components of a remote system can devote substantial computing resourcesto processing audio data, enabling more complex speech recognition andsemantic analysis functionality to be implemented than could otherwisebe implemented locally within a client device. However, a client-serverapproach necessarily requires that a client be online (i.e., incommunication with the remote systems) when processing voice input. Invarious situations, continuous online connectivity may not be guaranteedat all times and in all locations, so a client-server voice-based userinterface may be disabled in a client device whenever that device is“offline” and thus unconnected to an online service. Further, aclient-server approach can consume significant bandwidth, as it requirestransmission of high-bandwidth audio data from a client to components ofa remote system. The consumption of bandwidth is amplified in thetypical situations where the remote system is handling requests from alarge quantity of client devices. Yet further, a client-server approachcan exhibit significant latency in rendering of responses to a user,which can cause voice-based user-client interactions to be protracted,and resources of the client device to be utilized fora protractedduration. The latency of the client-server approach can be the result oftransmission delays, and/or delays in the voice-to-text processing,semantic processing, and/or response generation performed by remotesystem. Yet further, exchange of messages between client and server in aclient-server approach may require a relatively significant amount ofpower consumption for the transmission and reception of the messages.The effect of this may be particularly felt by the client device, whoseavailable power is often provided by on-device batteries with relativelylimited storage capacity.

SUMMARY

Implementations disclosed herein relate to generating and/or utilizing,by a client device, a text-response map that is stored locally on theclient device. The text-response map can include a plurality ofmappings, where each of the mappings define a corresponding directrelationship between corresponding text and a corresponding response.Each of the mappings is defined in the text-response map based on thecorresponding text being previously generated from previous audio datacaptured by the client device and based on the corresponding responsebeing previously received from a remote system in response totransmitting, to the remote system, at least one of the previous audiodata and the corresponding text.

When a spoken utterance is detected by the client device, the clientdevice can utilize a local voice-to-text/speech-to-text (STT) model togenerate text that corresponds to the spoken utterance. The clientdevice can then determine whether the generated text matches any of thecorresponding texts of the text-response map. If so, and optionally ifone or more other conditions are satisfied (e.g., condition(s) based ona confidence score described herein), the client device can utilize acorresponding mapping of the text-response map to select the responsethat has the direct relationship to the corresponding text, as definedby the corresponding mapping. The client device can then immediatelyrender the response, from its local memory, via one or more outputdevices of the client device (e.g., speaker(s) and/or display(s)). Theresponse is rendered responsive to the spoken utterance, and renderingthe response can include rendering text of the response, graphic(s) ofthe response, audio data of the response (or audio data converted fromthe response using a locally stored text-to-speech (TTS) processor),and/or other content. The response can additionally or alternativelyinclude a command to be transmitted by the client device, such as acommand transmitted (e.g., via WiFi and/or Bluetooth) to one or moreperipheral devices to control the peripheral device(s). As explainedbelow, having determined that the text generated by the localvoice-to-text/speech-to-text (STT) matches any of the correspondingtexts of the text-response map, the response may be provided by theclient device without the client device needing to transmit dataindicative of the detected spoken utterance to the remote system.

In these and other manners, the client device can render the responsewithout necessitating performance of, and/or without awaitingperformance of, local and/or remote resource intensive and latencyinducing: semantic processing of the text to determine a meaning orintent of the text; and generation of the response based on thedetermined meaning or intent. Accordingly, when the generated textmatches one of the corresponding texts of the text-response map, theresponse that has the direct relationship to the text (as defined by acorresponding mapping of the text-response map) can be rendered withreduced latency and/or with reduced resource consumption. Moreover, invarious situations the response can be rendered without bandwidthconsuming transmission, to a remote system, of audio data (that capturesthe spoken utterance) or text generated based on the audio data. Thismay further improve the battery-life of the client device, or otherwisefree-up power resources for other tasks at the client device, aspower-intensive transmission and reception of messages to/from theremote system is reduced.

In some implementations, a method is provided that includes the steps ofthe client device capturing audio data of a spoken utterance of the userand processing the audio data to generate text. In such implementations,the STT processing is performed locally on the client device and doesnot require that the audio data be submitted to the cloud. Next, atext-response mapping is accessed, which includes text mapped toresponses. This mapping is constructed based on prior texts generatedfrom audio data of prior spoken utterances of the user mapped to theresponses that were received from the remote system when the audio datawas previously submitted. The client device then determines whether thetext-response mapping includes a text mapping for the generated text. Inresponse to determining that the text-response mapping includes a textmapping that matches the text, the mapped response for that text mappingis selected and rendered by the client device.

If the text is not included in the mapping, the audio data (and/orclient-device generated text that corresponds to the audio data) issubmitted to the remote system for further processing. STT and/or NLP isperformed by the remote system to determine an action that correspondsto the spoken utterance, the generated action is utilized to generate aresponse, and the response is provided to the client device. The clientdevice can then render the response as well as store the generated textwith the response. When the user subsequently submits the same spokenutterance, the client device can locally process the audio data togenerate corresponding text, check that the text is included in themapping, and render the mapped response without requiring NLP processingand submission to the server. Thus, not only does the local mapping onthe client device save computational time, the method may be performedoffline if the mapping already includes the text.

In some instances, responses are dynamic and may result in differentresponses for the same spoken utterances. For example, a spokenutterance of “What time is it right now” is a dynamic query which varieseach time it is submitted. Other examples of dynamic queries includeweather, queries related to the location of the user, and other queriesthat are time-sensitive. On the other hand, some queries can be staticand rarely, if ever, result in a different response for a givenutterance. For example, “What is the capital of the United States” willalways result in the same response regardless of when the utterance issubmitted. In some instances, a response may be static for a givenperiod of time and then expire. For example, “What is the weather today”will likely remain static for the duration of the day and subsequentlyexpire at a given time, such as at midnight.

In some implementations, to assist in keeping mapped responses for textsstored locally on the client device fresh, the client device may submitthe audio data or other data indicative of the spoken utterance, such asclient-device generated text that corresponds to the audio data, to theremote system even when the text is identified in the mapping. Theclient device can provide the mapped response and once a response isreceived from the remote system, the received response can be checkedwith the mapped response. If the mapped response matches the receivedresponse, the client device can update the mapping to reflect that theresponse is static. For example, a confidence score that is associatedwith each text mapping may be updated to reflect that the same responsewas received as is stored in the mapping. If a different response wasreceived, the confidence score may be updated to reflect that the storedand received responses do not match. For example, in some instances, ifthe responses do not match, the text mapping may be removed to ensurethat the stale response is not provided to the user subsequently. Insome instances, the mapping may be flagged as stale without removing thetext mapping. In some instances, the mapped response may be providedonly if the confidence score associated with the mapping satisfies athreshold. For example, the server response may be provided a requisitenumber of times (with each server response being the same) before theconfidence score for the mapped response reaches a level at which themapped response is provided in lieu of providing the audio data, orother data indicative of the spoken utterance, to the remote system.Optionally, when the confidence score satisfies the threshold the clientdevice may automatically not transmit the audio data, or other dataindicative of the spoken utterance, to the remote system. Alternatively,in implementations where the client device does transmit the audio data,or other data indicative of the spoken utterance, to the remote system,the client device may provide the mapped response from its local memorybefore receiving a reply from the remote system.

In some implementations, a method implemented by one or more processorsof a client device is provided and includes capturing, via at least onemicrophone of the client device, audio data that captures a spokenutterance of a user. The method further includes processing the audiodata to generate current text that corresponds to the spoken utterance.Processing the audio data to generate the current text utilizes avoice-to-text model stored locally on the client device. The methodfurther includes accessing a text-response map stored locally on theclient device. The text-response map includes a plurality of mappings,where each of the mappings define a corresponding direct relationshipbetween corresponding text and a corresponding response based on thecorresponding text being previously generated from previous audio datacaptured by the client device and based on the corresponding responsebeing previously received from a remote system in response totransmitting, to the remote system, at least one of the previous audiodata and the corresponding text. The method further includes determiningwhether any of the corresponding texts of the text-response map matchesthe current text. The method further includes, in response todetermining that a given text, of the corresponding texts of thetext-response map, matches the current text: selecting a given responseof the corresponding responses of the text-response map, and causing thegiven response to be rendered via one or more user interface outputdevices associated with the client device. Selecting the given responseis based on the text-response map including a mapping, of the mappings,that defines the given response as having a direct relationship with thegiven text.

These and other implementations of the technology can include one ormore of the following features.

In some implementations, the method further includes: transmitting theaudio data or the current text to the remote system; receiving, from theremote system in response to transmitting the audio data or the currenttext, a server response that is responsive to the spoken utterance;comparing the server response to the given response; and updating thetext-response map based on the comparison. In some versions of thoseimplementations, receiving the server response occurs after at leastpart of the given response has been rendered via the one or more userinterface output devices. In some additional or alternative versions ofthose implementations, comparing the server response to the givenresponse indicates that the server response differs from the givenresponse. In some of those additional or alternative versions, updatingthe text-response map includes, based on the comparison indicating thatthe server response differs from the given response: updating themapping, that defines the given response as having the directrelationship with the given text, to define the server response ashaving the direct relationship with the given text. In some of thoseadditional or alternative versions, updating the text-response mapincludes, based on the comparison indicating that the server responsediffers from the given response: removing, from the text-response map,the mapping that defines the given response as having the directrelationship with the given text. In some of those additional oralternative versions, updating the text-response map includes, based onthe comparison indicating that the server response differs from thegiven response: storing, in the text-response map, data that preventsthe given text from being mapped to any responses.

In some implementations that include updating the text-response map,updating the text-response map includes adjusting a confidence scoreassociated with the mapping that defines the given response as havingthe direct relationship with the given text. In some versions of thoseimplementations, adjusting the confidence score associated with themapping that defines the given response as having the directrelationship with the given text includes: adjusting the confidencescore to be more indicative of confidence if the comparison indicatesthe given response matches the server response. In some additional oralternative versions of those implementations, selecting the givenresponse is further based on the confidence score associated with themapping satisfying a threshold.

In some implementations, the method further includes: capturing, via theat least one microphone of the client device, additional audio data thatcaptures an additional spoken utterance; processing, utilizing thevoice-to-text model stored locally on the client device, the additionalaudio data to generate additional text that corresponds to theadditional spoken utterance; determining whether any of thecorresponding texts of the text-response map matches the additionaltext; and in response to determining that none of the correspondingtexts of the text-response map matches the additional text: transmittingat least one of the additional text and the additional audio data to theserver system, receiving, from the server system in response totransmitting the at least one of the additional text and the additionalaudio data, an additional response, and causing the additional responseto be rendered via one or more of the user interface output devicesassociated with the client device. In some of those implementations, themethod further includes: receiving, from the server system with theadditional response, an indication that the server response is a staticresponse for the additional text; and in response to receiving theindication that the server response is a static response for theadditional text: adding, to the text-response map, a new mapping thatdefines a new direct relationship between the additional text and theadditional response.

In some implementations, the client device lacks any connection to theInternet when the method is performed.

In some implementations, the method further includes determining aconfidence score associated with the mapping that defines the givenresponse as having the direct relationship with the given text. In someof those implementations, causing the given response to be renderedincludes: causing, in response to the confidence score satisfying athreshold, the given response to be rendered without transmitting theaudio data or the current text to the remote system.

In some implementations, the method further includes: transmitting,prior to the given response being rendered, the audio data or thecurrent text to the remote system; determining a confidence scoreassociated with the mapping that defines the given response as havingthe direct relationship with the given text; and determining, based onthe confidence score, a threshold amount of time to await receiving,from the remote system in response to transmitting the audio data or thecurrent text, a server response that is responsive to the spokenutterance. In some of those implementations, causing the given responseto be rendered includes: causing the given response to be rendered atexpiration of the threshold amount of time when the server response isnot received before expiration of the threshold amount of time.

In some implementations, a method implemented by one or more processorsof a client device is provided and includes: capturing, via at least onemicrophone of the client device, audio data that captures a spokenutterance of a user; and processing the audio data to generate currenttext that corresponds to the spoken utterance. Processing the audio datato generate the current text utilizes a voice-to-text model storedlocally on the client device. The method further includes accessing atext-response map stored locally on the client device. The text-responsemap includes a plurality of mappings, and each of the mappings define acorresponding direct relationship between corresponding text and acorresponding response based on the corresponding text being previouslygenerated from previous audio data captured by the client device andbased on the corresponding response being previously received from aremote system in response to transmitting, to the remote system, atleast one of the previous audio data and the corresponding text. Themethod further includes: determining, by the client device, that thecorresponding texts of the text-response map fail to match the currenttext; transmitting, to a remote system, the audio data or the currenttext; receiving, from the remote system in response to submitting theaudio data or the current text, a response; and updating thetext-response map by adding a given text mapping. The given text mappingdefines a direct relationship between the current text and the response.The method further includes: capturing, subsequent to updating thetext-response map, second audio data; processing the second audio datato generate a second text utilizing the voice-to-text model storedlocally on the client device; determining, based on the text-responsemap, that the current text matches the second text; and in response todetermining that the current text matches the second text, and based onthe text-response map including the given text mapping that defines thedirect relationship between the current text and the response: causingthe response to be rendered via one or more user output devicesassociated with the client device.

These and other implementations of the technology can optionally includeone or more of the following features.

In some implementations, the method further includes receiving, with theresponse, an indication of whether the response is static. In some ofthose implementations: adding the given text mapping to thetext-response map occurs in response to the indication indicating thatthe response is static.

In some implementations, updating the text-response map further includesstoring a confidence score in association with the given text mapping,where the confidence score is indicative of likelihood that the responseis static. In some of those implementations, the method furtherincludes: submitting the second audio data to the remote system;receiving, in response to submitting the second audio data, a secondserver response from the remote system; and further updating theconfidence score based on the second server response.

In some implementations, the method further includes: receiving, withthe response, an indication that the response is static only until anexpiration event occurs; updating the text-response map to include anindication of the expiration event with the given text mapping; andremoving the given text mapping from the text-response map when theexpiration event occurs.

In some implementations, updating the text-response map includesremoving one or more mappings from the text-response map.

Some implementations include a computing apparatus including one or moreprocessors and at least one memory storing computer-executableinstructions which, when executed by the one or more processors, causesthe one or more processors to perform a method, such as a methoddescribed above or elsewhere herein. The computing apparatus can be, forexample, a client device. The one or more processors can include, forexample, central processing unit(s), graphics processing unit(s), and/ortensor processing unit(s). Some implementations include a non-transitorycomputer readable medium including computer-executable instructionswhich, when executed by one or more processors of at least one computingapparatus, cause a method to be performed, such as a method describedabove or elsewhere herein.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example environment in which techniques describedherein can be implemented.

FIG. 2 provides a flowchart that illustrates implementations of methodsdescribed herein.

FIG. 3 is a flowchart that illustrates a method of verifying that atext-response map includes a static response for a text.

FIG. 4 illustrates a flowchart of an example method according toimplementations disclosed herein.

FIG. 5 illustrates a flowchart of another example method according toimplementations disclosed herein.

FIG. 6 is a block diagram of an example computing device that mayoptionally be utilized to perform one or more aspects of techniquesdescribed herein.

DETAILED DESCRIPTION

In the implementations discussed hereinafter, a semantic processor of avoice-enabled client device utilizes a text-response map stored locallyon the client device to parse spoken utterances received by the device.In some implementations, the text-response map is generated based onprevious spoken utterances received by the device and one or moreresponses received from a cloud-enabled device. In instances where thetext-response map does not include a text corresponding to the receivedspoken utterance, the device can provide audio data that captures thespoken utterance (and/or a text representation of the spoken utterancegenerated by the client device) to the cloud-based device, which maythen perform further analysis, determine a response, and provide theresponse to the client device for rendering by one or more interfaces ofthe device. The response can then be stored with text in thetext-response mapping stored locally on the client device and utilizedto identify a response upon future instances of receiving the samespoken utterance. As outlined above and explained further below, thiscan lead to more efficient use of hardware resources in a networkinvolving the client device and the remote, cloud-enabled device.

Further details regarding selected implementations are discussedhereinafter. It will be appreciated however that other implementationsare contemplated so the implementations disclosed herein are notexclusive.

Referring to FIG. 1, an example environment in which techniquesdescribed herein can be implemented is illustrated. The exampleenvironment includes a client device 100 and a remote system 150. Theclient device 100 may be, for example: a desktop computing device, alaptop computing device, a tablet computing device, a mobile phonecomputing device, a computing device of a vehicle of the user (e.g., anin-vehicle communications system, an in-vehicle entertainment system, anin-vehicle navigation system), a standalone interactive speaker, a smartappliance such as a smart television, and/or a wearable apparatus of theuser that includes a computing device (e.g., a watch of the user havinga computing device, glasses of the user having a computing device, avirtual or augmented reality computing device). Additional and/oralternative client devices may be provided. Components of client device100 and components of remote system 150 can communicate viacommunication network 101. Communication network 101 may include, forexample, a wide area network (WAN) (e.g., the Internet). Further,components of client device 100 may communicate with one or more othercomponents via communication network 101. For example, communicationnetwork 101 may include a local area network (LAN) and/or BLUETOOTH, andmay communicate with one or more other devices via the LAN and/orBLUETOOTH.

Client device 100 includes one or more microphones 106 that may captureaudio data indicative of one or more spoken utterances of a user. Themicrophone 106 may then provide the audio data to one or more othercomponents of client device 100 and/or remote system 150 for furtherprocessing.

Client device 100 may include a number of modules suitable forimplementing the herein-described methods, including, for example, aspeech-to-tech (STT) module 102, a mapping module 103, a remotecomponent module 104 and a render module 105, as well as a text-responsemap 107 for storing a plurality of text mappings. Likewise, remotesystem 150 may include a number of modules, including, for example, aremote STT module 151, a natural language processing (NLP) module 152,and an agent engine 153 suitable for interacting with one or more agents190.

Referring now to FIG. 2, and with continued reference to FIG. 1, aflowchart is provided that illustrates implementations of methodsdescribed herein using the various components illustrated in FIG. 1. Asillustrated, audio data 200 may be provided to STT module 102. STTmodule 102 receives audio data 200 and converts the digital audio datainto one or more text words or phrases (also referred to herein astokens). In some implementations, STT module 102 can be a streamingmodule, such that audio data of captured utterances is converted to texton a token-by-token basis and in real time or near-real time, such thattokens may be output from STT module 102 effectively concurrently with auser's speech, and thus prior to a user enunciating a complete spokenrequest. STT module 102 may rely on one or more locally-stored offlineacoustic and/or language models, which together model a relationshipbetween an audio signal and phonetic units in a language, along withword sequences in the language. In some implementations, a single modelmay be used, while in other implementations, multiple models may besupported, e.g., to support multiple languages, multiple speakers, etc.

In some instances, audio data 200 and/or text 205 may be provided toremote component module 104. Remote component module 104 communicateswith remote system 150 via communication network 101 and may provide theaudio data and/or a text representation of the audio data to remote STTmodule 151 and/or natural language processing (NLP) module 152. RemoteSTT module 151 can function similar to STT module 102 in that it mayreceive audio data indicative of a spoken utterance of a user andconvert the audio data into text. However, remote STT module 151 doesnot utilize the resources of client device 100 and instead utilizes theresources of remote system 150. Thus, in some instances, remote STTmodule 151 may be more robust that STT module 151 because remote system150 has less constraints on computing power and/or storage than clientdevice 100.

Text generated by STT module 102 and/or by remote STT module 151 may beprovided to NLP module 152 for further processing. NLP module 152processes free form natural language input and generates, based on thenatural language input, annotated output for use by one or more othercomponents of the remote system 150. For example, the NLP module 152 canprocess natural language free-form input that is textual input that is aconversion, by STT module 151 and/or remote STT module 151, of audiodata provided by a user via client device 106. Also, for example, NLPmodule 152 may generate output directly from the audio data 200 receivedfrom remote component module 104. The generated annotated output mayinclude one or more annotations of the natural language input andoptionally one or more (e.g., all) of the terms of the natural languageinput.

In some implementations, the NLP module 152 is configured to identifyand annotate various types of grammatical information in naturallanguage input. For example, the NLP module 152 may include a part ofspeech tagger (not depicted) configured to annotate terms with theirgrammatical roles. Also, for example, in some implementations the NLPmodule 152 may additionally and/or alternatively include a dependencyparser (not depicted) configured to determine syntactic relationshipsbetween terms in natural language input.

In some implementations, the NLP module 152 may additionally and/oralternatively include an entity tagger (not depicted) configured toannotate entity references in one or more segments such as references topeople (including, for instance, literary characters, celebrities,public figures, etc.), organizations, locations (real and imaginary),and so forth. The entity tagger of the NLP module 152 may annotatereferences to an entity at a high level of granularity (e.g., to enableidentification of all references to an entity class such as people)and/or a lower level of granularity (e.g., to enable identification ofall references to a particular entity such as a particular person). Theentity tagger may rely on content of the natural language input toresolve a particular entity and/or may optionally communicate with aknowledge graph or other entity database to resolve a particular entity.Identified entities may be utilized to identify patterns in the text, asdescribed herein.

In some implementations, the NLP module 152 may additionally and/oralternatively include a coreference resolver (not depicted) configuredto group, or “cluster,” references to the same entity based on one ormore contextual cues. For example, the coreference resolver may beutilized to resolve the term “there” to “Hypothetical Café” in thenatural language input “I liked Hypothetical Café last time we atethere.”

In some implementations, one or more components of the NLP module 152may rely on annotations from one or more other components of the NLPmodule 152. For example, in some implementations the named entity taggermay rely on annotations from the coreference resolver and/or dependencyparser in annotating all mentions to a particular entity. Also, forexample, in some implementations the coreference resolver may rely onannotations from the dependency parser in clustering references to thesame entity. In some implementations, in processing a particular naturallanguage input, one or more components of the NLP module 152 may userelated prior input and/or other related data outside of the particularnatural language input to determine one or more annotations.

NLP module 152 may then provide the annotated output to agent module153, which may provide the output to one or more agents 190, which maythen determine an appropriate response for the spoken utterance of theuser. For example, agent module 153 may determine, based on theannotations of the output of NLP module 152, which agent 190 is mostlikely to utilize the annotated output and generate a meaningfulresponse. The annotated output of the NLP module 152 can then beprovided to that agent and the resulting response from the agent may beprovided to the client device 100 for rendering. For example, a spokenutterance of “What is the weather today” may be converted to text,annotated by NLP module 152, and the resulting annotated output may beprovided to agent module 153. Agent module 153 may determine, based onthe annotation, that a weather agent may likely determine a meaningfulresponse to the annotated output, and provide the output to the weathercomponent. The weather agent may then determine a response to provide tothe client device, which may then be rendered by the client device 100to the user. For example, based on the response of “75, sunny” providedby agent module 153 from a weather agent, rendering module 105 mayprovide audio and/or visual rendering of the response (e.g., an audiorendering of “The weather will be sunny today and 75” and/or output to avisual interface of the weather). The NLP module 152 may be relativelycomputationally demanding when processing the text generated by STTmodule 102 and/or by remote STT module 151. This means that the NLPmodule's presence at the remote system 150 is advantageous to the clientdevice 100 and system as a whole because, whilst the client device 100may be computationally capable of implementing the NLP module 152, itsoverall processing capabilities and power-storage capacity are likely tobe more limited than those available at the remote system 150. Thesefactors mean that the NLP module's presence at the remote system 150keeps latency of response low, particularly where the client device 100needs to wait for natural language processing in order to provide theresponse, and keeps general reliability of response high regardless ofwhether the eventual response is provided from the mapped local storageat the client device or by waiting for a response from the remote system150.

Once a response has been received by agent module 153 from one or moreagents 190, the response may be provided to the client device 100 viaremote component module 104. The received response may be provided torendering module 105, which may then render the action via one or morecomponents (not illustrated). For example, render module 105 may includea text-to-speech component, which may convert the response into speechand provide the audio to the user via one or more speakers of the clientdevice 100. Also, for example, render module 105 may generate agraphical response and provide the graphical response via one or morevisual interfaces associated with the client device 100. Also, forexample, render module 105 may be in communication with one or moreother devices of the user, such as Wi-fi controlled lighting, and mayprovide the response to the one or more other devices for rendering(e.g., turn on a light).

In some implementations, the server response can be stored intext-response map 107 with the text 200. Text-response map 107 includestexts generated from audio data by STT module 102 and/or remote STTmodule 151, stored with corresponding responses received from the remotesystem 150. For example, audio data for a spoken utterance of “Turn onthe kitchen light” may be received by STT module 102 and converted totext. The text may be provided to the remote system 150, which maydetermine an appropriate action for the command (e.g., determine themeaning of the spoken utterance). A resulting response indicative ofturning on a light may be received by render module 105, which then canidentify a “kitchen light” associated with the client device 100 andturn the light off. The resulting response can then be stored intext-response map 107 with the response (e.g., a response of turning offa particular light) for later utilization by the client device 100.

In some implementations, the text-response map may be stored as one ormore tables in a database, as a stack (e.g., a last-in first-out datastructure), as a queue (e.g., a first-in first-out data structure),and/or one or more alternative data structures. As described herein, thetext can be stored in an alphanumerical format. However, the text mayalternatively be stored as one or more matrices of phonemes, as audiodata, and/or any other format that allows mapping module 103 to compareaudio data captured by the client device 100 with text stored in thetext-response map. The responses stored in the text-response module maybe stored as a textual response, an action to be performed by one ormore interfaces of the user, and/or one or more alternative formats thatcan be provided to one or more interfaces for rendering to the user.

The text and response are stored in the text-response map with a directrelationship such that each text is associated with a particularresponse. For example, the text “What is the capital of France” can bestored with a response of “Paris” and with no other responses. Also, forexample, the text “Turn on the kitchen light” can be stored with adirect relationship with a command that can be provided to a lightingdevice of the user to turn on the device.

In some instances, the text and response may already be stored in thetext-response map 107. For example, the user may have already spoken theutterance “Turn off the kitchen light” and the resulting response mayhave already been stored with the text. In those instances, mappingmodule 103 can verify that the result received from the remote system150 matches the response stored in the text-response map. Mapping module103 may update the text-response map 107 based on checking whether thestored response matches the response received from the remote system150.

Referring again to FIG. 2, mapping module 103 can access thetext-response map 107 to determine whether the generated text 205 hasbeen previously stored, as described above. At decision 210, if a textmapping for text 205 is not identified in the text-response mapping 107,the text (or the audio data) is provided to the remote system 150 and aresponse 220 is received from the remote server 150, as previouslydescribed. The response 220 can then be rendered by render module 105and further provided to mapping module 103, which then generates a textmapping based on the text 205 and the response 220. The text mapping canthen be stored in text-response map 107.

If the mapping module 103 determines at decision 210 that text-responsemap 107 includes a text mapping for the text 205, the mapping module 103accesses the text-response map 107, identifies the response 215associated with the text 205, and provides the response 215 to rendermodule 105 for rendering to the user. Thus, in instances where a spokenutterance of the user has already been processed by the remote system150 and stored in the text-response map 107, the remote system 150 doesnot need to be accessed to render content to the user. Because allactions occur on the client device 100, any latency introduced bycommunication network 101 and/or by components of remote system 150 iseliminated. Furthermore, overall power consumption is reduced due to thefact that the remote system 150 does not need to be accessed by theclient device 100. Further, because natural language processing anddetermining a response are eliminated, local and/or remote computationresources are saved when a user submits the same utterance repeatedly.Still further, because the response can be rendered without utilizingthe remote system 150, the response can be rendered even when the clientdevice 100 lacks any connection to the remote system 150, such as anInternet connection.

In some instances, a response can be a static response that does notchange between repeated requests by a user. For example, a spokenutterance of “Who was the first president of the United States” willresult in the same response each time the utterance is received.However, in some instances, responses can be dynamic and may change uponreceiving the same spoken utterance. For example, “What restaurants arenear me” will change with the location of the user and is therefore adynamic response. Also, for example, “What is the weather tomorrow” isdynamic in that the weather predictions may change throughout the dayand further, “tomorrow” describes a particular day for a limited amountof time (i.e., only when the utterance is received “today”).

In some implementations, remote component module 104 provides the text205 and/or the audio data 200 to remote system 150 even when mappingmodule 103 identifies a text mapping for text 200 in the text-responsemap 107. This can occur, for example, before the render module 105 isprovided the corresponding response of the text mapping or render module105 may be provided the response (and start the rendering of theresponse) before receiving a server response. In some implementations,mapping module 103 may wait a threshold amount of time after remotecomponent engine 104 provides the text and/or audio data and onlyprovide the render module 105 with the locally stored response if aresponse is not received from the remote system 150 before expiration ofthe threshold time.

Referring to FIG. 3, a flowchart is provided that illustrates a methodof verifying that the text-response map 107 includes a static responsefor a text. STT module 102 can provide the text 205 to the mappingmodule 103, which then can identify a text mapping for the text 200 anda corresponding response 205. Further, STT module can provide the text205 (or the audio data 200) to remote component module 104, whichreceives a response 215 from the remote system 150. Mapping module 103can then compare the corresponding response 215 with the server response220 at decision 300. If the corresponding response 215 matches theserver response 220, the corresponding response is more likely to bestatic 305 and the text mapping may be updated accordingly. However, ifthe corresponding response 215 does not match the server response 220,the corresponding response 215 is likely to be dynamic 310 and the textmapping for the text 205 can be updated accordingly.

In some implementations, mapping module 103 may store a confidence scorewith text mappings. The confidence score may be indicative of likelihoodthat the corresponding response is static. The confidence score can beupdated each time a server response matches the corresponding responseto reflect a greater likelihood that the response for the text will notchange subsequently. For example, a confidence score of “1” may beassigned to a text mapping when it is first stored in the text-responsemap 107. Subsequently, mapping module 103 may identify the text mapping(based on a subsequent text that matches the text of the text mapping),the text (or audio data) may be provided to the remote system 150, and aserver response received from the remote system may be compared to thecorresponding response. If the two responses match, the confidence scorefor the text mapping may be updated to “2” to reflect that the sameresponse was identified twice. As the text is processed again insubsequent submission(s) from the user, the confidence score maycontinue to be updated.

In some implementations, a corresponding response of a text mapping thatis identified by mapping module 103 may be provided to render module 105only if the confidence score associated with the identified text mappingsatisfies a threshold. In these circumstances, the render module 105 mayrender the response without waiting for any further response from theremote system 150. This reduces latency. For example, mapping module 103may identify a text mapping with a confidence score of “3,” indicatingthat the corresponding response has been verified three times. Mappingmodule may only provide the corresponding text if the confidence levelis greater than “2” and may provide the corresponding text. Also, forexample, the mapping module 103 may not provide a corresponding responseif the associated confidence of the text mapping is “1.” Instead, remotecomponent module can provide the audio data 200 and/or text 205 to theremote system 150 and the server response can be provided to the rendermodule 105 (and to the mapping module 103 to update the confidence scoreas described above). If the corresponding response matches the serverresponse, the confidence score may be updated to “2” and, when the textmapping is identified based on a subsequent utterance of the user, thecorresponding response from the mapping may be provided to the rendermodule 105 instead of the server response.

In some implementations, remote component module may provide the audiodata 200 and/or text 205 to the remote system 150 only when mappingmodule 103 does not identify a text mapping in text-response map 107 (aspreviously described) or when the confidence score associated with anidentified text mapping does not satisfy a threshold. For example, acorresponding response for a text mapping with a confidence score of “3”may be provided to render module 105, without any communication with theremote server 150 if the confidence score satisfies a threshold.However, a corresponding response for a text mapping with an associatedconfidence score of “2” may be provided to the render module 105 andfurther, the audio data or text can be provided to the remote system150. In these circumstances, the response from the mapping locallystored at the client device 100 may be rendered before the response fromthe remote system 150 is received at the client device 100. When theresponse from the remote system 150 is subsequently received, it may beused to iterate the confidence score or, for example, remove the storedmapping, mark the stored mapping as stale and/or to update the mappingto reflect the fact that the stored response is dynamic as describedbelow. This depends on whether the response from the remote system 150matches the response in the local mapping. Thus, the remote server 150may only be accessed when a confidence score of an identified textmapping does not satisfy a threshold. Therefore, the resources of theremote server 150 are impacted only when confidence in a response storedon the client device 100 is not high enough to be assured that the textmapping includes a static response.

In some instances, the server response may not match the correspondingresponse of text mapping identified by mapping module 103. In thoseinstances, mapping module 103 may update the text mapping to reflectthat the stored response is dynamic. For example, mapping module 103 mayupdate the confidence score associated with the text mapping to “−1” orsome other flag to indicate that the response is stale and/or isdynamic. If the same text is subsequently generated by the STT module102, mapping module 103 can identify the text mapping, determine thatthe corresponding response should not be rendered, and instead indicateto the remote component module 104 to provide the audio data and/or textto the remote server 150 for further processing. By setting theconfidence score (or setting a flag) to reflect a response is dynamicensures that, upon subsequent instances of receiving the same text, anew mapping is not stored. Thus, mapping module 103 will not repeatedlyexpend computational resources continuously adding text mappings thatwill not ever be utilized to render content to the user. However, insome implementations, mapping module 103 may remove the text mappingentirely to preserve storage space.

In some implementations, remote system 150 may provide an indication ofwhether a server response is dynamic or static. In some instances, agentmodule 153 may determine an indication to provide to the client device100 based on the agent 190 to which the agent module 153 provided theaction based on the received audio data and/or text. For example, agentmodule 153 may determine that the action should be provided to a weatheragent to determine the weather for a particular day. Further, agentmodule 153 may determine that responses from the weather agent aredynamic and provide an indication that the server response provided tothe client device 100 is dynamic. Based on receiving an indication thatthe server response is dynamic, mapping module 103 may not store theserver response with the text in the text-response map 107 and/or maystore an indication with the text mapping indicating that the responseis dynamic and should not be served from the text-response map 107 uponsubsequent processing of the text. As another example, agent module 153may determine that a knowledge graph agent should process the annotatedtext and provide an indication that the knowledge graph utilized by thecomponent is static and that mapping module 103 should store the textmapping (i.e., the text with the server response) for future utilizationwhen the same spoken utterance is captured.

In some implementations, remote system 150 may provide an indication ofhow long a server response is static. For example, agent module 153and/or one or more other components of the remote system 150 maydetermine, once processed, that an utterance of “What is the weatherlike tomorrow” can result in a static response until midnight and thenwill be different after midnight (i.e., when the definition of“tomorrow” changes to a different day). Also, for example, an utteranceof “Who do the Cubs play next” may be provided with an indication of thetime of the next Cubs game and that the response will change after theCubs have played their next game. Also, for example, an utterance of“Find restaurants near me” may result in a static response only when theuser has not changed locations and/or has not moved more than athreshold distance between providing the utterances. Mapping module 103may then check to determine whether an expiration event stored with thetext mapping has occurred before providing the corresponding response tothe render module 105.

In some implementations, mapping module 103 may periodically remove textmappings from the text-response map 107 to ensure that the storedresponses are still fresh and/or to prevent the text-response map 107from growing in storage space requirements beyond the capabilities ofthe client device 100. For example, mapping module 103 may utilize a“first in, first out” approach and remove older text mappings that havenot been accessed when new text mappings are added (e.g., only keep thelast X number of accessed mappings and remove the one accessed thelongest time ago when a new mapping is added and the text-responseincludes X text mappings). In some implementations, mapping module 103may remove text mappings stored with an expiration event when the eventoccurs. For example, mapping module 103 may periodically checkexpiration events stored in the text-response map 107 and remove anytext mappings with expiration events that have already occurred. In someimplementations, mapping module 103 may periodically remove any textmappings that have been flagged as dynamic.

FIG. 4 illustrates a flowchart of an example method according toimplementations disclosed herein. One or more steps may be omitted,performed in a different order, and/or one or more additional steps maybe included in various implementations.

At step 405, audio data is captured that is indicative of a spokenutterance of a user. The spoken utterance may be captured by one or morecomponents of a client device that shares one or more characteristicswith microphone 106 of client device 100. In some implementations,portions of the spoken utterance may be captured before the user hascompleted the spoken utterance and the portions may be provided to oneor more other components while still capturing additional audio data.For example, microphone 106 may capture a portion of audio data andprovide the audio data to one or more components while continuing tocapture additional audio data of the spoken utterance.

At step 410, a current text is generated from the audio data. Thecurrent text may be generated by a component that shares one or morecharacteristics with STT module 102. In some implementations, STT module102 may receive a portion of the audio data and begin generating thecurrent text before the entirety of the audio data for the spokenutterance has been received from, for example, microphone 106. In someimplementations, STT module 102 may wait until all audio data of aspoken utterance has been provided before generating the current text.Further, STT module 102 may perform some normalization of the text to,for example, remove filler words, conjugate verbs to a standardconjugation, and/or remove unmeaningful portions of the text. However,STT module 102 is intended to be less computationally intensive than,for example, a STT module executing on a server. Thus, STT module 102may perform little more than conversion of the audio data to text.

At step 415, a text-response map is accessed. The text-response map mayshare one or more characteristics with text-response map 107 and may beaccessed by a component that shares one or more characteristics withmapping module 103. The text-response map includes text mappings, eachof which includes a text with a direct relationship to a correspondingresponse. The text mappings in the text-response map may be generatedbased on previous spoken utterances of the user and responses receivedfrom a remote system in response to submitting the audio data and/ortext generated in response to capturing the audio data of the userspeaking an utterance.

At step 420, the text-response map is checked to determine whether thecurrent text is included in a text mapping. For example, a componentsharing one or more characteristics with mapping module 103 may accessthe text-response map and determine whether the text of any of the textmappings includes the current text. In some implementations, mappingmodule 103 may require an exact match between the text of a text mappingand the current text. In some implementations, mapping module 103 mayidentify a close match to the current text. However, because the mappingmodule 103 is executing on a client device of the user and may beresource-constrained, mapping module 103 may not identify a given textin the text-response map as matching the current text unless the matchis exact or the texts vary minimally.

At step 425, once a text mapping with a text that matches the currenttext has been identified in the text-response map, the correspondingresponse for that text mapping is selected. The corresponding responsefor a given text may have been previously generated and stored in thetext-response map based on the given text (or audio data associated withthe given text) being submitted to a remote system and receiving thecorresponding response from the remote server. In some implementations,a confidence score may be associated with the text mapping and thecorresponding response may be selected only if the confidence scoresatisfies a threshold. The confidence score may be determined based on,for example, the number of times the given text has been submitted to aremote system and the corresponding response being received from theremote system. Thus, as a given text (or audio data for the given text)is provided to the remote system for processing with the same resultingresponse, the greater the confidence that the corresponding response isvalid.

At step 430, one or more components causes the response from theidentified text mapping to be rendered. The response may be rendered bya component that shares one or more characteristics with render module105. In some implementations, render module 103 may be in communicationwith one or more other components, such as a text-to-speech module whichconverts the response into speech and provides the speech to user viaone or more speakers. Also, for example, render module 105 may be incommunication with one or more other interfaces, such as a visualinterface, and rendering may include providing visual output to the uservia the visual interface. Also, for example, render module 103 may be incommunication with one or more other devices of the user, such as alighting fixture that is Wi-fi enabled, and cause the device to performone or more actions (e.g., turning off a particular light).

FIG. 5 illustrates a flowchart of another example method according toimplementations disclosed herein. One or more steps may be omitted,performed in a different order, and/or one or more additional steps maybe included in various implementations.

At step 505, one or more components captures audio data of a spokenutterance of a user. Step 505 may share one or more characteristics withstep 405 of FIG. 4. For example, the audio data may be captured by acomponent that shares one or more characteristics with microphone 106.

At step 510, a current text is generated from the audio data. This stepmay share one or more characteristics with step 410 of FIG. 4. Forexample, a component sharing one or more characteristics with STT module102 may generate the text based on the audio data.

At step 515, a text-response map is accessed. The text-response map mayshare one or more characteristics with the text-response map 107 andthis step may share one or more characteristics with step 415 of FIG. 4.For example, mapping module 103 may access the text-response map 107 todetermine whether the current text is included in a text mapping in thetext-response map 107.

At step 520, one or more components determines that the text-responsemap does not include a text mapping with a text that matches the currenttext. The determination may be performed by a component that shares oneor more characteristics with mapping module 103 in text-response map107. For example, mapping module 107 may access the text-response mapand check the stored texts in the map and determine that none of thetext mappings match the current text, none of the text mappings thatmatch have a confidence score indicative of valid data, and/or anymatching text mappings have expired, as described herein.

At step 525, the captured audio data and/or the current text areprovided to a remote system. The remote system may share one or morecharacteristics with remote system 159. For example, a remote componentmodule 104 may provide the audio data and/or text to remote system 150for further processing, as described herein. The remote system 150 maythen determine a response for the audio data and/or current text.

At step 530, the response is then provided by the remote system to theclient device. In some implementations, the response may be receivedwith an indication of the agent utilized by the remote system togenerate the response. In some implementations, the response may bereceived with an indication of whether the response is static. Ifinstead the response is received with an indication that the response isdynamic, the response may be rendered but not stored in thetext-response map according to the remaining steps of the method of FIG.5.

At step 535, the text-response map is updated. Updating thetext-response map may include generating a new text mapping thatincludes the current text mapped to the server response. In someimplementations, updating the text-response map may include storing aconfidence score with the new text mapping and/or an indication receivedwith the response (e.g., whether the response is static or dynamic, theagent utilized to generate the response).

At step 540, second audio data is captured. Step 540 may share one ormore characteristics with step 505 and/or step 405 of FIG. 4.

At step 545, a second text is generated from the second audio data. Step545 may share one or more characteristics with step 510 and/or step 410of FIG. 4.

At step 550, one or more components determines that the current textmatches the second text. This may be determined by a component thatshares one or more characteristics with mapping module 103 and can shareone or more characteristics with step 420 of FIG. 4. For example,mapping module 103 may determine that the text of the text mappingstored in the text-response mapping at step 535 matches the second text.

At step 555, one or more components causes the response to be rendered.This step may share one or more characteristics with step 430 of FIG. 4.For example, a component sharing one or more characteristics with rendermodule 105 can cause the response to be rendered. Optionally, therendering of the response may occur without the second audio data ordata representing the second audio data being sent to the remote system150.

FIG. 6 is a block diagram of an example computing device 610 that mayoptionally be utilized to perform one or more aspects of techniquesdescribed herein. For example, client device 100 can include one or morecomponents of example computing device 610 and/or one or more remotesystems 150 can include one or more components of example computingdevice 610.

Computing device 610 typically includes at least one processor 614 whichcommunicates with a number of peripheral devices via bus subsystem 612.These peripheral devices may include a storage subsystem 624, including,for example, a memory subsystem 625 and a file storage subsystem 626,user interface output devices 620, user interface input devices 622, anda network interface subsystem 616. The input and output devices allowuser interaction with computing device 610. Network interface subsystem616 provides an interface to outside networks and is coupled tocorresponding interface devices in other computing devices.

User interface input devices 622 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computing device 610 or onto a communication network.

User interface output devices 620 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a visible image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computing device 610 to the user or to another machine or computingdevice.

Storage subsystem 624 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 624 may include the logic toperform selected aspects of the methods of FIGS. 2-5, and/or toimplement various components depicted in FIGS. 1-3.

These software modules are generally executed by processor 614 alone orin combination with other processors. Memory 625 used in the storagesubsystem 624 can include a number of memories including a main randomaccess memory (RAM) 630 for storage of instructions and data duringprogram execution and a read only memory (ROM) 632 in which fixedinstructions are stored. A file storage subsystem 626 can providepersistent storage for program and data files, and may include a harddisk drive, a floppy disk drive along with associated removable media, aCD-ROM drive, an optical drive, or removable media cartridges. Themodules implementing the functionality of certain implementations may bestored by file storage subsystem 626 in the storage subsystem 624, or inother machines accessible by the processor(s) 614.

Bus subsystem 612 provides a mechanism for letting the variouscomponents and subsystems of computing device 610 communicate with eachother as intended. Although bus subsystem 612 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computing device 610 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computing device 610depicted in FIG. 6 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputing device 610 are possible having more or fewer components thanthe computing device depicted in FIG. 6.

In situations in which certain implementations discussed herein maycollect or use personal information about users (e.g., user dataextracted from other electronic communications, information about auser's social network, a user's location, a user's time, a user'sbiometric information, and a user's activities and demographicinformation, relationships between users, etc.), users are provided withone or more opportunities to control whether information is collected,whether the personal information is stored, whether the personalinformation is used, and how the information is collected about theuser, stored and used. That is, the systems and methods discussed hereincollect, store and/or use user personal information only upon receivingexplicit authorization from the relevant users to do so.

For example, a user is provided with control over whether programs orfeatures collect user information about that particular user or otherusers relevant to the program or feature. Each user for which personalinformation is to be collected is presented with one or more options toallow control over the information collection relevant to that user, toprovide permission or authorization as to whether the information iscollected and as to which portions of the information are to becollected. For example, users can be provided with one or more suchcontrol options over a communication network. In addition, certain datamay be treated in one or more ways before it is stored or used so thatpersonally identifiable information is removed. As one example, a user'sidentity may be treated so that no personally identifiable informationcan be determined. As another example, a user's geographic location maybe generalized to a larger region so that the user's particular locationcannot be determined.

1. A method implemented by one or more processors of a client device,the method comprising: capturing, via at least one microphone of theclient device, audio data that captures a spoken utterance of a user;processing the audio data to generate current text that corresponds tothe spoken utterance, wherein processing the audio data to generate thecurrent text utilizes a voice-to-text model stored locally on the clientdevice; accessing a text-response map stored locally on the clientdevice, wherein the text-response map includes a plurality of mappings,each of the mappings defining a corresponding direct relationshipbetween corresponding text and a corresponding response based on thecorresponding text being previously generated from previous audio datacaptured by the client device and based on the corresponding responsebeing previously received from a remote system in response totransmitting, to the remote system, at least one of the previous audiodata and the corresponding text; determining, by the client device, thatthe corresponding texts of the text-response map fail to match thecurrent text; transmitting, to a remote system, the audio data or thecurrent text; receiving, from the remote system in response tosubmitting the audio data or the current text, a response; updating thetext-response map by adding a given text mapping, the given text mappingdefining a direct relationship between the current text and theresponse; capturing, subsequent to updating the text-response map,second audio data; processing the second audio data to generate a secondtext utilizing the voice-to-text model stored locally on the clientdevice; determining, based on the text-response map, that the currenttext matches the second text; and in response to determining that thecurrent text matches the second text, and based on the text-response mapincluding the given text mapping that defines the direct relationshipbetween the current text and the response: causing the response, fromthe text-response map, to be implemented.
 2. The method of claim 1,further comprising: receiving, with the response, an indication ofwhether the response is static; and wherein adding the given textmapping to the text-response map occurs in response to the indicationindicating that the response is static.
 3. The method of claim 1,wherein updating the text-response map further comprises: storing aconfidence score in association with the given text mapping; wherein theconfidence score is indicative of likelihood that the response isstatic.
 4. The method of claim 3, further comprising: submitting thesecond audio data to the remote system; receiving, in response tosubmitting the second audio data, a second server response from theremote system; and updating the confidence score based on the secondserver response.
 5. The method of claim 1, further comprising:receiving, with the response, an indication that the response is staticonly until an expiration event occurs; updating the text-response map toinclude an indication of the expiration event with the given textmapping; and removing the given text mapping from the text-response mapwhen the expiration event occurs.
 6. The method of claim 1, whereinupdating the text-response map includes removing one or more mappingsfrom the text-response map.
 7. The method of claim 1, wherein the spokenutterance comprise a query or command and the response comprises aresponse to the query or command.
 8. The method of claim 7, wherein theresponse comprises information indicated to be of interest in the queryor command.
 9. The method of claim 1, wherein, in the text-response mapstored locally on the client device, one or more of the mappingscomprises: a corresponding mapping between a corresponding textcomprising a query or command and a corresponding response to said queryor command.
 10. The method of claim 1, wherein the correspondingresponses in the text-response map stored locally on the client devicecomprise information indicated to be of interest in the correspondingtexts.
 11. A client device comprising: one or more processors;microphones; one or more user output devices; and memory storingcomputer-executable instructions which, when executed by the one or moreprocessors, causes the one or more processors to perform a methodcomprising: capturing, via the microphones, audio data that captures aspoken utterance of a user; processing the audio data to generatecurrent text that corresponds to the spoken utterance, whereinprocessing the audio data to generate the current text utilizes avoice-to-text model stored locally on the client device; accessing atext-response map stored locally on the client device, wherein thetext-response map includes a plurality of mappings, each of the mappingsdefining a corresponding direct relationship between corresponding textand a corresponding response based on the corresponding text beingpreviously generated from previous audio data captured by the clientdevice and based on the corresponding response being previously receivedfrom a remote system in response to transmitting, to the remote system,at least one of the previous audio data and the corresponding text;determining that the corresponding texts of the text-response map failto match the current text; transmitting, to a remote system, the audiodata or the current text; receiving, from the remote system in responseto submitting the audio data or the current text, a response; updatingthe text-response map by adding a given text mapping, the given textmapping defining a direct relationship between the current text and theresponse; capturing, subsequent to updating the text-response map,second audio data; processing the second audio data to generate a secondtext utilizing the voice-to-text model stored locally on the clientdevice; determining, based on the text-response map, that the currenttext matches the second text; and in response to determining that thecurrent text matches the second text, and based on the text-response mapincluding the given text mapping that defines the direct relationshipbetween the current text and the response: causing the response, fromthe text-response map, to be implemented.
 12. The client device of claim11, wherein the response comprises a command and wherein causing theresponse to be implemented comprises causing the command to be providedto a smart device to alter a state of the smart device.
 13. The clientdevice of claim 12, wherein the smart device is a lighting device. 14.The client device of claim 11, wherein the method further comprises:receiving, with the response, an indication of whether the response isstatic; and wherein adding the given text mapping to the text-responsemap occurs in response to the indication indicating that the response isstatic.
 15. The client device of claim 11, wherein updating thetext-response map further comprises: storing a confidence score inassociation with the given text mapping; wherein the confidence score isindicative of likelihood that the response is static.
 16. The clientdevice of claim 11, wherein the method further comprises: submitting thesecond audio data to the remote system; receiving, in response tosubmitting the second audio data, a second server response from theremote system; and updating the confidence score based on the secondserver response.
 17. The client device of claim 11, wherein the methodfurther comprises: receiving, with the response, an indication that theresponse is static only until an expiration event occurs; updating thetext-response map to include an indication of the expiration event withthe given text mapping; and removing the given text mapping from thetext-response map when the expiration event occurs.
 18. The clientdevice of claim 11, wherein causing the response to be implementedcomprises causing at least one of the output devices to render theresponse.